Evidence for the Interpretation and Use of Scores from an Automated Essay Scorer

This paper examined validity evidence for the scores based on the Intelligent Essay Assessor (IEA), an automated essay-scoring engine developed by Pearson Knowledge Technologies. A study was carried out using the validity framework described by Yang, et al. (2002). This framework delineates three approaches to validation studies: examine the relationship among scores given to the same essays by different scorers, examine the relationship between essay scores and external measures, and examine the scoring processes used by the IEA. The results of this study indicated that, although relationships among scores given to the same essays by different scorers (percent agreement, Spearmen rank-order correlation, kappa statistic and Pearson correlation) indicated a stronger relationship between two human readers than between the IEA and a human reader, stronger relationships were found between the IEA and experts than between readers and experts. In addition, the results of examining the scoring processes used by the IEA showed that the IEA used processes similar to a human scorer. Furthermore, the IEA scoring processes were more similar to processes used by proficient human scorers than to processes used by non-proficient or intermediate human scorers. The results of this study provided positive evidence for the use of IEA scores as measures of writing achievement. Further research with the IEA in other assessments and grade levels will be helpful in generalizing the results of this study and further strengthening the validity of IEA for scoring writing assessments.